Convergence to a simplex ETF: Class means tend towards equinorm and equiangular vectors when centred about the global average.

Throughout this paper, we assume that the entries of W are subgaussian.

"juntas" [Blum and Langley, 1997]), that is functions that depend only on a small number

The remarkable capability of over-parameterised neural networks to generalise effectively has been explained by invoking a "simplicity bias": neural networks prevent overfitting by initially learning simple classifiers before progressing to

Symmetries are prevalent in deep learning and can significantly influence the learning dynamics of neural networks. In this paper, we examine how exponential symmetries - a broad subclass of continuous symmetries present in the model architecture or loss function - interplay with stochastic gradient descent (SGD). We first prove that gradient noise creates a systematic motion (a "Noether flow") of the parameters θ along the degenerate direction to a unique initialization-independent fixed point θ

Our main results are summarized below.

However, such a global learning rate schedule does not take into account the structural characteristics of neural networks (NNs).